{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a script to monitor how many valid data points for yesterday's recorded weather are returned by each of our forecasters. The script saves the plot everyt 60 seconds. To find out what time the x axis column corresponds to, use the `what_time` function defined below. This analysis was performed on September 23 for the date of September 22 so if you want to reproduce the results, just set `yesterday_string` variable to `2018-09-22`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import datetime\n",
    "import time\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first series of while loops and try blocks are there to make sure we are analyzing a csv file that is whole. The current script for ETL of the recorded data updates the csv file continuously so the retreival of its data is messed up while the ETL script saves it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEKCAYAAAAcgp5RAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl8VNX9//HXyWQnGwkhEBJI2EnC\nHgWhIIIbghta/FpbUamKP2xduohLW/Xbb6v9+a0/bdUW91KrVEEFFK0iuAEKRLaQKGFPCGQjC5ms\nM+f3x9yEBAnZ7sydmXyej0ceM3PnztzPXODNyZlzz1Faa4QQQvivAKsLEEII4V4S9EII4eck6IUQ\nws9J0AshhJ+ToBdCCD8nQS+EEH5Ogl4IIfycBL0QQvg5CXohhPBzgVYXANCnTx+dkpJidRlCCOFT\ntm3bVqK1jm9vP68I+pSUFLZu3Wp1GUII4VOUUoc6sp903QghhJ+ToBdCCD8nQS+EEH7OK/roz6Sh\noYH8/Hxqa2utLkWYLDQ0lKSkJIKCgqwuRYgewWuDPj8/n8jISFJSUlBKWV2OMInWmtLSUvLz80lN\nTbW6HCF6BK/tuqmtrSUuLk5C3s8opYiLi5Pf1ITwIK8NekBC3k/Jn6sQnuW1XTdC9EgHv4D9n1pd\nhTBD4jgYOcfqKgAJeiG8R91J+PeNYC8F5Lce36YhJBqWHAIv+A1Wgv4sDh48yNy5c9m9e3er7b/9\n7W+ZPn06F154YZuvffjhh4mIiOCXv/zlWY9RWlrKrFmzADh27Bg2m434eNcVzRdccAGDBg3i7rvv\nBuCSSy4hOTmZF154AYBf/OIXDBgwgHnz5pGamsrTTz/Nz372MwDuvPNOMjMzuemmm7r02btrw4YN\nBAcHM2XKFEuO75O2PO8K+YUfQ/I5VlcjumPzc/DBEqgugYh2ZyhwO6/uo/dWjz766FlDvjPi4uLY\nvn0727dvZ9GiRdxzzz3Nj6dOncrGjRsBcDqdlJSUkJ2d3fzajRs3Ngdp3759eeqpp6ivrzelru7a\nsGFDc+2iA+qq4MunYeiFEvL+IHaI67Zsn7V1GHyiRf/I6mz2HK009T3TEqP43eXp7e7ncDi49dZb\n2bhxIwMGDODdd9/ljjvuYO7cuVx77bW8//773HvvvfTq1YupU6eyf/9+1qxZA8CePXuYMWMGhw8f\n5u677+bnP/95p2qcMmUK99xzDwDZ2dlkZGRQWFjIiRMnCA8PJycnhwkTJnD06FHi4+OZOnUqr776\nKrfeemu7752Xl8eiRYsoLi7GZrPx5ptvMnjwYH7961+zdu1alFI89NBDXHfddWzYsIEnnnii+XO1\n/G0hJSWFBQsWsHr1ahoaGnjzzTcJDQ3lb3/7GzabjX/+85/85S9/Ydq0aZ367D3O10uhpgxmPGB1\nJcIMcUbQl+6DgZOtrQUfCXor7d27l9dff53nn3+e+fPns2LFiubnamtruf322/nss89ITU3l+uuv\nb/Xa3Nxc1q9fT1VVFSNGjOCOO+7o1EVCiYmJBAYGcvjwYTZu3Mh5551HQUEBmzZtIjo6mtGjRxMc\nHNy8/3333cfs2bO55ZZb2n3vG264gSVLlnD11VdTW1uL0+lk5cqVbN++nR07dlBSUsI555zD9OnT\n232vPn36kJWVxbPPPssTTzzBCy+8wKJFizrUdSWA2krY+BcYdjEkTbS6GmGGmEEQEAileVZXAvhI\n0Hek5e0uqampjBs3DoCJEydy8ODB5udyc3MZPHhw84U/119/PUuXLm1+fs6cOYSEhBASEkLfvn05\nfvw4SUlJnTr+lClT2LhxIxs3buTee++loKCAjRs3Eh0dzdSpU1vtO3jwYCZNmsS//vWvs75nVVUV\nBQUFXH311YDrSlWAL774guuvvx6bzUZCQgLnn38+W7ZsISoq6qzvN2/ePMB1flauXNmpzycwWvMn\nYMYSqysRZrEFusLeS7pupI++HSEhIc33bTYbjY2NHnltk6Z++l27dpGRkcHkyZPZtGlTq/75lh54\n4AEef/xxtNadPlZbAgMDcTqdzY9Pv9ip6XN29TP2aM2t+UtggLTm/UrcECjdb3UVgAR9t4wYMYL9\n+/c3t/KXL19u+jGmTJnCmjVriI2NxWazERsbS3l5OZs2bTpj0I8cOZK0tDRWr17d5ntGRkaSlJTE\nO++8A0BdXR12u51p06axfPlyHA4HxcXFfPbZZ5x77rkMGjSIPXv2UFdXR3l5OevWrWu37sjISKqq\nqrr+wXuKr/4OteXSmvdHcUNdLXoTG11dJUHfDWFhYTz77LNceumlTJw4kcjISKKjo009xujRoykp\nKWHy5MmttkVHR9OnT58zvubBBx8kPz//rO+7bNkynn76acaMGcOUKVM4duwYV199NWPGjGHs2LHM\nnDmTP/3pT/Tr14/k5GTmz59PRkYG8+fPZ/z48e3Wffnll/P2228zbtw4Pv/888596J6itgI2/QWG\nz4YBE6yuRpgtdjA02KGq0OpKUGb+it9VmZmZ+vQVpnJychg1apRFFXXcyZMniYiIQGvN4sWLGTZs\nWPNIGdE2X/nzdasNj8OGP8Btn7quohT+Zd8nsOxqWLAGUt0z6kwptU1rndneftKi76bnn3+ecePG\nkZ6eTkVFBbfffrvVJQlfUFMOm5+BEXMk5P1V3FDXrRd8IesTo2682T333NPhFnzLq2BbWrduHXFx\ncWaXBsDixYv58ssvW2276667uPnmm91yPNFBX/3N1XUz4z6rKxHuEpUEthCvGGIpQe9BTVfBetIz\nzzzj0eOJDqgph03Pwsi50H+s1dUIdwkIgNhUrxh5I103Qnja5mehrkJG2vQEsUO8outGgl4IT6o5\n4ZrwatTl0G+01dUId4sbAmUHwOmwtAwJeiE8adOzUFcJ50trvkeIGwKOOqg4+3Bnd5OgF8JT7GVG\na/4K6JdhdTXCE7xkFksJ+rM4ePAgGRnf/wf529/+lo8//visr3344Yd54oknOnwsm83GuHHjmn8e\ne+wxABoaGliyZAnDhg1jwoQJnHfeeaxdu7ZzH8REr7zyCkePHrXs+D5t0zNQXyV98z1J0xDLUmuD\nXkbddMGjjz5q+nuGhYWdcUTOb37zGwoLC9m9ezchISEcP36cTz+1bqm5V155hYyMDBITEy2rwSfZ\ny1zTHaRdBQnWTdInPCyyHwT1kqDvkLVL4Nguc9+z32iY/Vi7u1k5H73dbuf555/nwIEDzROHJSQk\nMH/+/DZf88EHH/DAAw/gcDjo06cP69ato6ysjFtuuYX9+/cTHh7O0qVLGTNmzPdWwcrIyGiuffbs\n2fzgBz9o9bnfe+89tm7dyg033EBYWBibNm0iLCysU5+pxyrY5mrNn/NTqysRnqSUayoE6brxbnv3\n7mXx4sVkZ2cTExNzxvno165dy7Zt2yguLm712tzcXD788EO+/vprHnnkERoaGto8Tk1NTauum+XL\nl5OXl8fAgQPbnSa4SXFxMbfeeisrVqxgx44dvPnmmwD87ne/Y/z48ezcuZM//OEP3HjjjV363Nde\ney2ZmZm89tprbN++XUK+M+ylrtso+U2ox4kbLC36DulAy9tdPDUf/Zm6bnbu3NmpWjdv3sz06dOb\n64mNjQVc88w3/Qc1c+ZMSktLqaw8+4pdZ/vcoguagj7cPVdACy8WNxRy3wNHA9g6vvCQmTrcoldK\n2ZRS3yil1hiPU5VSXyml8pRSy5VSwcb2EONxnvF8intK9wwr56MfOnQohw8fbjeUu+ps88ybMZe+\naMFeCsoGoebObip8QOwQcDZC+WHLSuhM181dQE6Lx48DT2qthwIngIXG9oXACWP7k8Z+fsnd89GH\nh4ezcOFC7rrrruZFv4uLi5u7ZE43efJkPvvsMw4cOABAWVkZANOmTeO1114DXIt29+nTh6ioKFJS\nUsjKygIgKyur+XVnI/PMd5G91NWaV8rqSoSntVw/1iIdCnqlVBIwB3jBeKyAmcBbxi6vAlcZ9680\nHmM8P8vY3++YOR/96X30S5a4huD9/ve/Jz4+nrS0NDIyMpg7d26bffbx8fEsXbqUefPmMXbsWK67\n7jrANdRz27ZtjBkzhiVLlvDqq64/nmuuuYaysjLS09P561//yvDhw9ut86abbmLRokWMGzeOmpqa\nLn3WHqkp6EXP4wWzWHZoPnql1FvAH4FI4JfATcBmo9WOUioZWKu1zlBK7QYu1VrnG8/tAyZprUva\nen+Zj77n8ZU/X9O8fBmg4Ob3rK5EeJrW8NggGPNDmPO/pr61afPRK6XmAkVa622mVHbqfW9TSm1V\nSm09fbSKL5H56EWH2EshPNbqKoQVlLJ85E1HRt1MBa5QSl0GhAJRwFNAjFIqUGvdCCQBBcb+BUAy\nkK+UCgSigdLT31RrvRRYCq4WfXc/iFWsmo9+0qRJ1NXVtdq2bNkyRo+WibK8knTd9GyxQyD/a8sO\n327Qa63vB+4HUErNAH6ptb5BKfUmcC3wBrAAeNd4ySrj8Sbj+U90F9cr1FrjT937Zs5H/9VXX5ny\nPlbwhuUrPcrpdF0ZK0Hfc8UNhd0roLEOAkPa399k3blg6j7gXqVUHhAHvGhsfxGIM7bfC3RpYo/Q\n0FBKS0t7Xij4Oa01paWlhIaGWl2K59SWg3ZI0PdkcUMA7Zqy2AKdumBKa70B2GDc3w+ce4Z9aoEf\ndrewpKQk8vPzv3e1qfB9oaGhbV445pfsrmGuEvQ9WMtZLPuO9PjhvfbK2KCgoOYrPIXwaXJVrIgb\n7Lq16AtZmetGCHdrCvpeEvQ9Vlhv13/0Fi0ULkEvhLtJi16AsX6sNQuFS9AL4W4S9AJcX8hK140Q\nfspeCoGhEBRudSXCSnFDoOoo1Fd7/NAS9EK4W9MYej+6JkR0QfPIG89333jtqBsh/IZMf+B1Gh1O\n7A0OauodVNc1Yq93YK930OBwtv/iDhoYG05ybIvf4lrOYtnPs1ewS9AL4W4y/YFHaa05XllHzrFK\ncguryD1WyXfHT1JZ04C93hXqdY3mBXpb+keHsnHJzFNX97ccS+9hEvRCuJu9BGLGW12FX3E6NbWN\nDqrrHByrqCXnWCU5haeC/YT91LKdA2LCGJ4QQVr/KMKDbYSH2AgPCqRXiI2wYBu9ggObb4NsypRp\nVz7ac4znPz9AUVUdCVHGVeAhERDRz5IvZCXohXA3adF3WoW9gf95fw9Hy2ubW+Gun1P3TxcWZGNE\nv0guzejHyH5RjOofxYh+kUSHWbN83/OfHyD7aMWpoAfLRt5I0AvhTo4GqK2QoO+ERoeTO1/PYvP+\nUkYPiCY8OJA+ESFGazyQ8CDjNthGr2AbfSJCGNk/ioGx4dgCvOML71H9IwHILqhk5siEU0/EDobv\nPvB4PRL0QrhTzQnXrQR9h/3h/Vw+31vCn64Zw/xzkq0up0siQ4MYFBdO9tHT1nuOGwrVxa7//D24\nfrAMrxTCneRiqU7599YjvPTlAW6emuKzId8kPTGK7MKK1hstWj9Wgl4Id5Kg77Bth8p46O3d/GBo\nHx68zPeXmUxPjOZIWQ0VNae+GLZqLL0EvRDuJEHfIUfLa7h9WRaJMaH89UfjCbT5fjSlJUYBkFPY\novsmNhVQ0qIXwq9I0Lerpt7Bbcu2Utvg4IUFmcSEB1tdkinSjaBv1U8fFAbRSR6fxVK+jBXCnZqD\nXq6MPROtNb96awfZRyt5cUEmQ/tGWl2SafpGhhIfGUL20dP66WMHe/yiKWnRC+FO9jIIjrRknVBf\n8OyGfazZWch9l45sPQzRT6QnRrHneyNvPD+WXoJeCHeqLpHWfBv+k32M//vht1w1LpHbpw+2uhy3\nSOsfxd6ik9Q2tLjAK26oax3hpiUmPUCCXgh3kqtiz+jbY1Xcs3w7Y5KieeyaMaZMO+CN0hOjcTg1\n3x2vOrWxaeSNB/vpJeiFcCcJ+u85UV3PT/+xhfCQQJb+JJPQIJvVJblN0xeyrbpvLBhLL1/GCuFO\n9jKIH2l1FZYpOVnXPNlY02ySeUUnQcHy2ybTLzq0/TfxYQNjw4kICWw98iZmECibR7+QlaAXwp3s\npdCrj9VVeITWmg+zj5N1+AQ5hZXkFFZRcrKu+fmEqBBG9oti2vA+XJzWj/EDe1tYrWcEBCjS+ke1\nHnkTGAwxAz3adSNBL4S7NNRAQ3WP+TL231uPcN+KXQQHBjAiIZILRsQzsn8Uo/pFMrJ/FLG9/GN8\nfGelJUaxfMsRHE59atI1D4+8kaAXwl2aRlX0gD76sup6/rg2l3NTYvnXrZP84spWs6QnRlHT4OBA\nSTVD+0a4NsYOgcObQWuPLDEpfxpCuEsPuir2j+/ncLK2kd9fnSEhf5r0RNcsla26b+KGQv1JOFnk\nkRrkT0QId+khQf/1gTLe3JbPT6cNZniC/1zZapahfSMIsin2tJzzJs64bsBD/fQS9EK4Sw8I+gaH\nk4fe2cWAmDB+Pmuo1eV4peDAAIYnRLYeYunh9WMl6IVwlx7QR//iFwf47vhJHrkinfBg+cqvLemJ\nUWQfrURr7doQMxACgjz2hawEvRDuYi8BFITGWF2JW+SfsPPUx3u5KC2BC9P8b54aM6UnRlNWXc+x\nylrXhgCba8pi6boRwsfZSyEsBmz+2dJ9eNUe1+0V6RZX4v2apywuOK37xkMLkEjQC+Eufjz9wX+y\nj/FxznHuvnAYA2LCrC7H643qH4VSp81NH2cEvdPp9uNL0AvhLn4a9Pb6Rh5ZvYcRCZHc8oNUq8vx\nCb1CAkmN63XaEMsh0FgLVUfdfnwJeiHcxV4G4f43/cFT6/ZSUF7D76/OIEjGzHfYqMSo1kMsPTiL\nZbt/SkqpUKXU10qpHUqpbKXUI8b2VKXUV0qpPKXUcqVUsLE9xHicZzyf4t6PIISXspf63fQH3x6r\n4sXPDzA/M4lzUvzrs7lbemIU+SdqqLAbi4V7cBbLjvx3XAfM1FqPBcYBlyqlJgOPA09qrYcCJ4CF\nxv4LgRPG9ieN/YToWbT2u64bp1Pz0Du7iAgNZMnsUVaX43Oar5AtNLpvIhPh2pdg2MVuP3a7Qa9d\nThoPg4wfDcwE3jK2vwpcZdy/0niM8fws5a+rCgjRlvqT4Kj3q6B/KyufLQdPcP/skT12grLu+N7c\n9AEBkHENxCS7/dgd6mBTStmUUtuBIuAjYB9QrrVuNHbJBwYY9wcARwCM5ysA//nbLkRH+NlVsSeq\n6/nj+zlkDurNDye6P5j8UZ+IEBKiQlqPvPGQDgW91tqhtR4HJAHnAt1eSUEpdZtSaqtSamtxcXF3\n304I7+JHQe9wah56ZzeVxqRlAQHyC3pXpSdGtx554yGd+spca10OrAfOA2KUUk1XgiQBBcb9AiAZ\nwHg+Gig9w3st1Vpnaq0z4+Pju1i+EF7KT6Y/0Frz4Nu7eG9XIb++ZAQj+0VZXZJPS0+MYl9xdevF\nwj2gI6Nu4pVSMcb9MOAiIAdX4F9r7LYAeNe4v8p4jPH8J7p5ggcheojqEtetD4+60VrzyOo9vLHl\nCD+bOZTbzx9idUk+L61/FA6n5ttjVe3vbKKOtOj7A+uVUjuBLcBHWus1wH3AvUqpPFx98C8a+78I\nxBnb7wWWmF+2EF7Ox7tutNY8/sG3vLLxID/9QSr3XjTc6pL8wqm56T3bT9/uJBxa653A+DNs34+r\nv/707bXAD02pTghfZS91LQAdGm11JV3y9Lo8/vbpPn48eSAPzhmFDJwzR3JsGJGhgR7vp5fL2oRw\nh6Yx9D4YkH//dB9Pfvwd105M4tErMiTkTaRU02Lhnm3RS9AL4Q72Uujle9MfvLrxIH9cm8vlYxN5\n/JoxMsLGDdITo8k9VonD6bmvLiXohXAHe5nP9c8v33KY363K5uK0BP48fyw2CXm3SE+MorbByf7i\nk+3vbBIJeiHcwcfmuXl3ewFLVu7i/OHx/OVH42WyMjdKH2BcIVvoue4b+dMUwh18aJ6btbsKufff\nO5icGsfffzKRkECb1SX5tSHxEQQHBni0n16CXgizOZ1Q4xtdNx/sLuTnb3zDuOQYXliQSWiQhLy7\nBdkCGJEQ6dGRNxL0Qpitthy00+uDfmVWPov/9Q2jB0Tz8s3n0CvEP5c89EbfWyzczSTohTCbD0x/\nsGzzIVd3zeBYli2cRFRokNUl9SjpiVGU2xs4WlHrkeNJ0AthNrt3T3/w90/38Zt3dnPhqL68uEBa\n8lZIa7pCtsAz3TcS9EKYzUunP9Ba87//+bZ5nPxzP54offIWGdU/8vuLhbuR/FcuhNm8MOi11jy6\nZg8vf3mQ6zKT+cO80TJO3kLhwYEM7tPLY0MsJeiFMFtz0HvHlbEOp+aBlbtYvvUIt0xN5TdzZe4a\nb5CeGM22Qyc8cizpuhHCbPZSCAyD4HCrK6HB4eSuN75h+dYj/HzmUAl5L5KWGEVBeQ0nquvdfiwJ\neiHM5iXTH9Q2OLjjn9tYs7OQ+2eP5N6LR0jIe5HmNWQ90H0jQS+E2bxk+oMH397NxzlF/PdVGbJo\niBdqmps+xwNBL330QpjNC6Y/KLfXs3rHURacN4ifTB5kaS3izGJ7BfP5ry8gqXeY248lLXohzOYF\nQb96ZyH1Dic/zEy2tA5xdsmx4R7pTpOgF8JsXhD0K7PyGZEQ2dwPLHo2CXohzORogNoKS4N+X/FJ\nvjlczjUTB8iXrwKQoBfCXM3z3Fj3ZezbWQUEKLhq3ADLahDeRYJeCDNZfFWs06l5+5sCpg2Lp29U\nqCU1CO8jQS+EmSwO+s0HSikor2HeBGnNi1Mk6IUwU1PQW7Qw+IptBUSGBHJJej9Lji+8kwS9EGay\nsEVvr29k7e5CLhvdX2alFK1I0AthpqYvY8M8/2Xsh9nHsNc7uGZiksePLbybBL0QZrKXQkgUBAZ7\n/NArthWQHBtG5qDeHj+28G4S9EKYyaJ5bgoravhyXwnzxicRIPPMi9NI0AthJouuin37mwK0Rkbb\niDOSoBfCTBYEvdaalVkFZA7qzaC4Xh49tvANEvRCmMmCoN+ZX0Fe0Un5Ela0SYJeCDNZEPQrs/IJ\nDgzgstH9PXpc4Tsk6IUwS70dGuwe/TK2vtHJqh1HuTgtgeiwII8dV/gWCXohzFLTNKGZ51r0678t\n4oS9gWsmSLeNaJsEvRBmab4q1nPTH6zMyqdPRAjThlkz5YLwDe0GvVIqWSm1Xim1RymVrZS6y9ge\nq5T6SCm117jtbWxXSqmnlVJ5SqmdSqkJ7v4QQngFD09/cKK6nk9yi7hqXCKBNmmzibZ15G9HI/AL\nrXUaMBlYrJRKA5YA67TWw4B1xmOA2cAw4+c24DnTqxbCG9k923WzeudRGhxaRtuIdrUb9FrrQq11\nlnG/CsgBBgBXAq8au70KXGXcvxL4h3bZDMQopWQ4gPB/Hm7Rr9iWz6j+UYzqL8sFirPr1O97SqkU\nYDzwFZCgtS40njoGJBj3BwBHWrws39gmhH+zlwIKwmLcfqi8oip25FdwjVwJKzqgw0GvlIoAVgB3\na60rWz6ntdaA7syBlVK3KaW2KqW2FhcXd+alQngneymE9YYA908RvCKrAFuA4kpZLlB0QIeCXikV\nhCvkX9NarzQ2H2/qkjFui4ztBUByi5cnGdta0Vov1Vpnaq0z4+Pju1q/EN7DQxdLOZyad74p4Pzh\n8cRHhrj9eML3dWTUjQJeBHK01n9u8dQqYIFxfwHwbovtNxqjbyYDFS26eITwX9UlHgn6lVn5FFbU\nygRmosMCO7DPVOAnwC6l1HZj2wPAY8C/lVILgUPAfOO594HLgDzADtxsasVCeCt7GfROceshDpVW\n8/CqbCalxjI7Q8Y4iI5pN+i11l8AbU1wPesM+2tgcTfrEsL32EthgPsuG2lwOLnrje3YAhRPXjcO\nm8w7LzqoIy16IUR7tHYFvRsXBX963V62HynnmR9NIDEmzG3HEf5HLqcTwgx1VeBscFsf/dcHynhm\nfR4/nJjEnDHSZSM6R4JeCDO48WKpipoG7lm+neTYcH53Rbrp7y/8n3TdCGEGN01/oLXmoXd2c7yy\nlrfumEJEiPyTFZ0nLXohzOCmFv3KrAJW7zjKPRcNZ1yy+6+4Ff5Jgl4IMzQHvXmLjhwqrea37+5m\nUmosi84fYtr7ip5Hgl4IM5jcopehlMJM0uEnhBnspRAQCCHmzCT5F2Mo5V9/NF6GUopukxa9EGaw\nG9MfqO63vL8+UMZf1+dx7cQk5o5JNKE40dNJ0AthBnuZKd02LYdSPixDKYVJpOtGCDOYNHPlY2tz\nOFZZywoZSilMJC16IcxgQtDXNzpZs6OQayYMkKGUwlQS9EKYwYSg33qwjKq6Ri4cldD+zkJ0ggS9\nEN3ldEDNiW4H/Se5RQQHBjB1qPsmRhM9kwS9EN1VWwHaaUrQTx4cRy/pmxcmk6AXortMuFjqQEk1\n+0uqmTWyr0lFCXGKBL0Q3WXC9Aef5LqWXJ4pQS/cQIJeiO4yoUW/PreIYX0jSI4NN6koIU6RoBei\nu6pLXLddDPqq2ga+OlAqrXnhNhL0QnRXN1v0X+wtocGhJeiF20jQC9Fd9lIIDIPgrnW7fJJbRFRo\nIBMH9Ta5MCFcJOiF6K5uzHPjdGrWf1vE+SP6EmiTf47CPWTArvAdBdugIMvqKr6vcAf06lrQ7yqo\noORkPTNHxptclBCnSNAL3+B0wOvXw8njVldyZmOu69LL1uUWEaDg/OHSPy/cR4Je+IZDX7pC/spn\nYfglVlfzfWFdG0O/PreI8QN7E9sr2OSChDhFgl74ht0rIagXpF/d5S89vU1RZS27Cir41SUjrC5F\n+Dn59kd4P0cj5KyCEZf6TcgDrP/WdTXsrFHSbSPcS4JeeL8Dn7qGMKbPs7oSU63LKSIxOpQRCZFW\nlyL8nAS98H7ZK12Lbg+90OpKTFPX6OCLvBJmjuqLMmGdWSHORoJeeLfGeshZDSMug6BQq6sxzVf7\ny7DXO+RqWOEREvTCu+37xDXfe8Y1Vldiqk9yiwgNCmDKEFlkRLifBL3wbtkrITQGBs+wuhLTaK1Z\nl3ucKUP6EBpks7oc0QNI0Avv1VALue/DqMsh0H/Gme8rPsmRshrpthEeI0EvvFfeR1BfBRn+Ndqm\naZGRCyTohYe0G/RKqZeUUkVBQHE3AAANTElEQVRKqd0ttsUqpT5SSu01bnsb25VS6mmlVJ5SaqdS\naoI7ixd+bvdKCO8DKdOtrsRU63KKGNkvkgExYVaXInqIjrToXwEuPW3bEmCd1noYsM54DDAbGGb8\n3AY8Z06Zosepr4bvPoC0K8DmPxdwV9Q0sPXQCem2ER7VbtBrrT8Dyk7bfCXwqnH/VeCqFtv/oV02\nAzFKqf5mFSt6kO8+gAa73422+ey7YhxOLVfDCo/qah99gta60Lh/DEgw7g8AjrTYL9/YJkTn7F4J\nEf1g4HlWV2KqT3KL6B0exLhkWWREeE63v4zVWmtAd/Z1SqnblFJblVJbi4uLu1uG8Ce1lbD3I0i/\nCgL8Z/ihw6nZ8G0RM0b0xRYgV8MKz+lq0B9v6pIxbouM7QVAcov9koxt36O1Xqq1ztRaZ8bHy6IL\nooVv14Kjzu/mttl+5AQn7A0y2kZ4XFeDfhWwwLi/AHi3xfYbjdE3k4GKFl08QnRM9kqISoKkc6yu\nxFSf5BZhC1CcP0waNsKz2h3OoJR6HZgB9FFK5QO/Ax4D/q2UWggcAuYbu78PXAbkAXbgZjfULPxZ\nzQnIWweTbocA/7rMY11OERMH9SY6PMjqUkQP027Qa62vb+OpWWfYVwOLu1uU6MFy3wNng9+Nttl7\nvIrcY1XcP3uk1aWIHsi/mkzC9+1eAb1TIHG81ZWY5su8En74901EhwUxZ4yMNhaeJ0EvvEd1Cez/\n1PUlrB/M0a615qUvDnDjS18THxHCu4unktTbf1bIEr7Dfy45FL4vZxVoh1/MbVPb4OChd3bz1rZ8\nLkpL4MnrxhERIv/chDXkb57wHrtXQtwwSMiwupJuKaqs5bZl29h+pJyfzxrG3bOGESDj5oWFJOiF\nd6g6Doe+hOm/8ulum+1Hyrl92Vaqaht57oYJzB4tffLCehL0wjvseRe006cvklqxLZ/7395F38gQ\nVtwxhVH9o6wuSQhAgl54i+yV0Dcd+vre8MNGh5M/rs3lxS8OcN7gOJ65YQKxvfxnoRTh+yTohfUq\n8uHwJpj5kNWVdMmv39rJym8KuGlKCg/OGUWQTQazCe8iQS+sl7PadeuD3Ta5xypZ+U0Bt08fzP2X\njbK6HCHOSJoewno5q10jbeKGWF1Jpz2zfh+9gm3cMcP3ahc9hwS9sNbJIji00bUAuI85UFLNezuP\n8uPJg4gJlz554b0k6IW1vn0f0D4Z9M9tyCPIFsDCaalWlyLEWUnQC2vtWQWxg6FvmtWVdEpBeQ0r\nswr4r3OS6RsZanU5QpyVBL2wTk05HPjU1Zr3sYukln66D4Dbzpe+eeH9JOiFdb77EJyNMOoKqyvp\nlKKqWl7fcoR5EwYwICbM6nKEaJcEvbBOziqITITECVZX0ikvfn6ARoeTO2YMtboUITpEgl5Yo77a\ntZLUqLk+tZJUub2ef24+xJwxiaT26WV1OUJ0iO/8CxP+JW8dNNb43Gibl788SHW9g8UXSN+88B0S\n9MIaOashPA4GTrG6kg47WdfIKxsPclFaAiP7yYRlwndI0AvPa6yH7z6AEZeBzXdm4fjn5kNU1DRw\n5wXSNy98iwS98LwDn0JdpU+NtqltcPDC5/uZNqwPY5NjrC5HiE6RoBeel7MKgiNh8PlWV9Jhb3x9\nmJKT9SyW1rzwQRL0wrOcDsh9D4ZfAoEhVlfTIfWNTv7+2X4yB/VmUmqs1eUI0WkS9MKzDm8Ce6lP\njbZ5+5t8CitquXPmUJSPXcErBEjQC0/LWQ2BoTD0Qqsr6ZBGh5PnNuxj9IBozh8eb3U5QnSJBL3w\nHK1dQT9kFoREWF1Nh7y3q5CDpXYWXzBEWvPCZ0nQC885mgWVBT7TbeN0ap5Zn8ewvhFcnNbP6nKE\n6DIJeuE5e1ZBQCCMuNTqStqltebPH33Hd8dP8n8uGEJAgLTmhe/ynatVhG/T2jWsMnU6hPW2upqz\n0lrz6Jo9vPzlQeZnJnHl2AFWlyREt0iLXnhGUQ6U7ff6bhuHU7NkxS5e/vIgN09N4bF5Y6Q1L3ye\ntOiFZ+SsBhSMmGN1JW1qcDi5Z/l21uws5Gczh3LvRcPlC1jhFyTohWfkrIaBkyEywepKzqi2wcGd\n/8ri45wilsweySJZOUr4Eem6Ee5Xth+O7/LabpvqukYWvrqFj3OK+O+rMiTkhd+RFr1wv5w1rtuR\nc62t4wwqahq4+eWv2X6knD/PH8u8CUlWlySE6dzSoldKXaqU+lYplaeUWuKOYwgfkrMa+o+F3oOs\nrqSV0pN1XL90M7sKKnj2hgkS8sJvmR70Sikb8AwwG0gDrldKpZl9HOEjKo9C/tde121zpMzO/L9v\nYl/xSZ6/MZNLM/pbXZIQbuOOrptzgTyt9X4ApdQbwJXAHjccS3i73Pdctx6ee97h1BRW1HC4zM7h\nUjuHjNvDZXYOlVZTWdtIREgg/7jlXCYNjvNobUJ4mjuCfgBwpMXjfGCSG44DWctg01/b3e1wmR2n\n1m4pQZxdnLOMsoBkfrrsGHDMI8esdzgpLK+l3uFs3hZkUyT1Dic5NpxxyTEMigtnxoi+DO3rG3Pu\nCNEdln0Zq5S6DbgNYODAgV17k/BYiB/R7m5lteUS9BYpI5WvIi9iWITnAtUWEMDsjDAGxYUzMNb1\nkxgThk0ufBI9lDuCvgBIbvE4ydjWitZ6KbAUIDMzs2spPHKO66cd47r05sIsE6wuQIgezh2jbrYA\nw5RSqUqpYOC/gFVuOI4QQogOML1Fr7VuVErdCXwI2ICXtNbZZh9HCCFEx7ilj15r/T7wvjveWwgh\nROfIFAhCCOHnJOiFEMLPSdALIYSfk6AXQgg/J0EvhBB+TmkvuGJUKVUMHLK6DqAPUGJ1Ed3gy/X7\ncu0g9VvJl2uH7tU/SGsd395OXhH03kIptVVrnWl1HV3ly/X7cu0g9VvJl2sHz9QvXTdCCOHnJOiF\nEMLPSdC3ttTqArrJl+v35dpB6reSL9cOHqhf+uiFEMLPSYteCCH8XI8PeqWUTSn1jVJqjfE4VSn1\nlbGw+XJjqmWvdIbaX1FKHVBKbTd+vHYqfqXUQaXULqPOrca2WKXUR0qpvcZtb6vrPJM2an9YKVXQ\n4txfZnWdbVFKxSil3lJK5SqlcpRS5/nQuT9T7T5x7pVSI1rUuF0pVamUutsT577HBz1wF5DT4vHj\nwJNa66HACWChJVV1zOm1A/xKaz3O+NluRVGdcIFRZ9PQsiXAOq31MGCd8dhbnV47uP7eNJ17b569\n9SngA631SGAsrr9DvnLuz1Q7+MC511p/21QjMBGwA2/jgXPfo4NeKZUEzAFeMB4rYCbwlrHLq8BV\n1lR3dqfX7ieuxHXOwYvPvS9TSkUD04EXAbTW9Vrrcnzg3J+ldl80C9intT6EB859jw564P8Bvwaa\nVpGOA8q11o3G43xci517o9Nrb/I/SqmdSqknlVIhFtTVURr4j1Jqm7F+MECC1rrQuH8MSLCmtHad\nqXaAO41z/5K3dn0AqUAx8LLR7feCUqoXvnHu26odfOPct/RfwOvGfbef+x4b9EqpuUCR1nqb1bV0\n1llqvx8YCZwDxAL3ebq2TviB1noCMBtYrJSa3vJJ7RoO5q1Dws5U+3PAEFxLFBcC/2thfWcTiGsZ\n3+e01uOBak7rKvDic99W7b5y7gEwvve7Anjz9Ofcde57bNADU4ErlFIHgTdwddk8BcQopZpW3jrj\nwuZe4Hu1K6X+qbUu1C51wMvAuVYWeTZa6wLjtghXP+W5wHGlVH8A47bIugrbdqbatdbHtdYOrbUT\neB7vPff5QL7W+ivj8Vu4wtMXzv0Za/ehc99kNpCltT5uPHb7ue+xQa+1vl9rnaS1TsH1a9QnWusb\ngPXAtcZuC4B3LSqxTW3U/uMWf1kUrn6+3RaW2SalVC+lVGTTfeBiXLWuwnXOwUvPfVu1N517w9V4\n6bnXWh8DjiilRhibZgF78IFz31btvnLuW7ieU9024IFz75Y1Y33cfcAbSqnfA99gfPHjI15TSsUD\nCtgOLLK4nrYkAG+7/j8iEPiX1voDpdQW4N9KqYW4ZjOdb2GNbWmr9mXGcFYNHARut67Edv0M19+V\nYGA/cDOuRp+3n3s4c+1P+8q5NxoHF9G6xsdw87mXK2OFEMLP9diuGyGE6Ckk6IUQws9J0AshhJ+T\noBdCCD8nQS+EEH5Ogl70eEqpjcZtilLqR1bXI4TZJOhFj6e1nmLcTQE6FfQtrqIWwmtJ0IseTyl1\n0rj7GDDNmCv8HuWa7///KqW2GBNm3W7sP0Mp9blSahWuq0qF8GrSGhHilCXAL7XWcwGMmSkrtNbn\nGDOBfqmU+o+x7wQgQ2t9wKJahegwCXoh2nYxMEYp1TT3UTQwDKgHvpaQF75Cgl6ItingZ1rrD1tt\nVGoGrilyhfAJ0kcvxClVQGSLxx8CdyilggCUUsNbLHQhhM+QFr0Qp+wEHEqpHcAruNYnSAGyjKmf\ni/HCJfaEaI/MXimEEH5Oum6EEMLPSdALIYSfk6AXQgg/J0EvhBB+ToJeCCH8nAS9EEL4OQl6IYTw\ncxL0Qgjh5/4/5G2DtuIkCicAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10add2080>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "count = 0\n",
    "while True:\n",
    "    \n",
    "    yesterday_string = (datetime.datetime.now().date()-datetime.timedelta(days=1)).strftime(\n",
    "        '{}-{}-{}'.format('%Y','%m','%d'))\n",
    "    \n",
    "    while True:\n",
    "        try:\n",
    "            df = pd.read_csv('/Users/Alex/Coding/weather_db/Data/TEST_history_db.csv')\n",
    "        except pd.errors.EmptyDataError:\n",
    "            continue\n",
    "            \n",
    "        csv_state = (df['date']==yesterday_string).count()\n",
    "        if csv_state >= count:\n",
    "            count = csv_state\n",
    "        else:\n",
    "            continue\n",
    "\n",
    "        df = df.set_index(['date','iter','provider'])\n",
    "        df = df.drop(index='2018-08-19',level=0)\n",
    "\n",
    "        try:\n",
    "            df_TWN = df.xs((yesterday_string,'TWN'), level=['date','provider'])\n",
    "            df_EC = df.xs((yesterday_string,'EC'), level=['date','provider'])\n",
    "            break\n",
    "        except KeyError:\n",
    "                continue\n",
    "    \n",
    "    df = df.reset_index()\n",
    "\n",
    "    TWN_count = df_TWN.groupby('iter').count()\n",
    "    EC_count = df_EC.groupby('iter').count()\n",
    "\n",
    "    success_prog = df.join(TWN_count, on='iter',how='outer',rsuffix='_TWN_count')\n",
    "    success_prog = success_prog.join(EC_count, on='iter',how='outer',rsuffix='_EC_count')\n",
    "\n",
    "    success_prog = success_prog[['iter','time','high_TWN_count','high_EC_count']].reset_index()\n",
    "\n",
    "    plot = success_prog.plot(x='iter',y=['high_TWN_count','high_EC_count'])\n",
    "    fig = plot.get_figure()\n",
    "    fig.savefig(\n",
    "        '/Users/Alex/Dropbox (Personal)/{}_history_collection.jpeg'.format(yesterday_string))\n",
    "    \n",
    "    # Comment out the following `break` line if continuous monitoring is required. \n",
    "    # Otherwise, it will analyze once only.\n",
    "    break\n",
    "    \n",
    "    time.sleep(60)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like Environment Canada (EC) reached a higher number of reported records than TWN and it also looks like they reached it earlier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's find out when:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def what_time(iter):\n",
    "    while True:    \n",
    "        try:\n",
    "            df = pd.read_csv('/Users/Alex/Coding/weather_db/Data/TEST_history_db.csv')\n",
    "            break\n",
    "        except pd.errors.EmptyDataError:\n",
    "            continue\n",
    "\n",
    "    return df[(df['date']==yesterday_string)&(df['iter']==iter)][['time']].iloc[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environment Canada was reporting upwards of 450 records around `iter` 56, which occured at:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "time    08:27:32.845741\n",
       "Name: 132000, dtype: object"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "what_time(56)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Meanwhile, The Weather Network never even reached 400 records and they peaked later, at `iter` 64, which occured at:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "time    13:05:37.619512\n",
       "Name: 141472, dtype: object"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "what_time(64)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
